Automatic Evaluation of Spoken Dialogue Systems

We advocate an objective evaluation methodology for the automated evaluation of spoken dialogue systems that eliminates manual interaction and reduces annotation errors and personal bias. The evaluation is performed by observing interactions between the system and a simulated user. We argue that user simulation is an inexpensive and feasible method for optimizing a dialogue system in the lab. Using a simulated user we can conduct dialogues until the performance measure reaches a predetermined conndence level. A simulated user not only exercises the dialogue system and points out defects, it also helps predict the success of a modiied dialogue strategy.